Skip to content

feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult - #732

Merged
zeroshade merged 20 commits into
apache:mainfrom
ennuite:gh-705-add-is-update-field
Aug 17, 2026
Merged

feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult#732
zeroshade merged 20 commits into
apache:mainfrom
ennuite:gh-705-add-is-update-field

Conversation

@ennuite

@ennuite ennuite commented Mar 27, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Following the discussion in the mailing list
, this PR introduces a new field in ActionCreatePreparedStatementResult, optional boolean is_update, that allows servers to indicate to clients the correct execution path for prepared statements.

What changes are included in this PR?

The new field is_update is added to PreparedStatement, as well as serialization/deserialization and getters.

FlightSql.pb.go was regenerated in order to include the changes in apache/arrow@2cd5ad2, with protoc-gen-go v1.36.11 and protoc v7.34.0.

Are these changes tested?

Yes. Unit tests for the new field were added to client_test.go, that complement the already existing unit tests for prepared statements queries (there were no tests for prepared statement updates). Integration tests for the new field were added to server_test.go. These are also the first integration tests for prepared statements.

Are there any user-facing changes?

Yes.
There are breaking changes to unkeyed literals for the exported structs flightsql.ActionCreatePreparedStatementResult and flightsql.CreatePreparedStatementResult.

There are also user-facing but non breaking changes: an extension to prepared statements was added to allow for reading/writing to the new field.

AI Disclaimer

This change was created with AI assistance (Augment Code and Claude Code and Codex). All lines were manually reviewed by a human. The output is not copyrightable subject matter.

Closes #705

@ennuite

ennuite commented Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

NB: I'm still investigating what led to such a large diff in the protobuf generated file. The changes correspond only to compiling the PR at apache/arrow#49498 with the following command:

cd /path/to/arrow-go/fork/arrow-go/arrow/flight

protoc --experimental_allow_proto3_optional \
  -I/path/to/arrow/fork/arrow/format \
  --go_out=./gen/flight \
  --go-grpc_out=./gen/flight \
  --go_opt=paths=source_relative \
  --go-grpc_opt=paths=source_relative \
  /path/to/arrow/fork/arrow/format/FlightSql.proto

@zeroshade am I supposed to merge changes to that file? I left them here for now to make the PR self contained for easier review, but I assume that if the PR in the main repo is merged, the generated code in this repo will be automatically updated. I will remove the changes if/when that time comes.

@ennuite ennuite changed the title feat(arrow/flight): Add is_update field to ActionCreatePreparedStatementResult feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult Mar 27, 2026
@zeroshade

Copy link
Copy Markdown
Member

NB: I'm still investigating what led to such a large diff in the protobuf generated file. The changes correspond only to compiling the PR at apache/arrow#49498 with the following command:

The diff is likely due to the fact that we haven't regenerated the protobuf code for quite a while, and protoc is now several versions ahead of when we last regenerated the code.

We also have a go generate that'll regenerate for you btw.

I don't consider the diff in the protobuf to necessarily be bad though

@zeroshade

Copy link
Copy Markdown
Member

I rebased this onto main which should solve the CI issues, I'll also re-review it

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the FlightSQL prepared statement API to surface a server-provided optional is_update hint (as *bool) so clients can choose the correct execution path (query vs update) when using prepared statements.

Changes:

  • Added IsUpdate *bool to the server-side ActionCreatePreparedStatementResult and mapped it into the protobuf action result.
  • Plumbed IsUpdate through the client prepared statement parsing/loading path and exposed it via PreparedStatement.IsUpdate().
  • Added unit tests (client) and integration tests (server) covering both is_update=false (query) and is_update=true (update) prepared statements.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.

File Description
arrow/flight/flightsql/server.go Adds IsUpdate to the server result struct and forwards it for CreatePreparedStatement responses (but currently not for Substrait plan responses).
arrow/flight/flightsql/client.go Stores/parses IsUpdate into PreparedStatement and exposes a getter; updates related documentation.
arrow/flight/flightsql/client_test.go Adds unit tests verifying the client reads IsUpdate and executes the appropriate code path.
arrow/flight/flightsql/server_test.go Adds integration tests (and server stubs) to validate end-to-end behavior for prepared statement query/update execution and the IsUpdate hint.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +137 to +153
query := req.GetQuery()
var isUpdate *bool
result.Handle = []byte(query)
switch query {
case "prepared query":
isUpdate := false
result.IsUpdate = &isUpdate
case "prepared update":
isUpdate := true
result.IsUpdate = &isUpdate
default:
err = fmt.Errorf("unknown query: %s", query)
}
if isUpdate != nil {
result.IsUpdate = isUpdate
}
return

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. This was also highlighted by Matt and I addressed it in #732 (comment)

Comment on lines 1290 to 1294
if output.ParameterSchema != nil {
result.ParameterSchema = flight.SerializeSchema(output.ParameterSchema, f.mem)
}
// is_update is not relevant for prepared substrait plans

@ennuite ennuite Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in fcde3da

Comment thread arrow/flight/flightsql/client.go Outdated
Comment on lines 1109 to 1111
// If the server returned the Dataset Schema or Parameter Binding schemas
// at creation, they will also be accessible from this object. Close
// or isUpdate at creation, they will also be accessible from this object. Close
// should be called when no longer needed.

@ennuite ennuite Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 0288679

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR and the thorough tests — the feature looks good.

(Edit: withdrawing my earlier concern about the regenerated FlightSql.pb.go toolchain — that's fine as-is, no need to regenerate.)

One thing to tidy before merge — see the inline comment on server_test.go: the isUpdate := … declarations inside the cases shadow the outer var isUpdate *bool, so the trailing if isUpdate != nil block is dead code. Dropping the outer var and that block keeps the helper correct and clear.

I also left an optional, non-blocking note on the IsUpdate() *bool getter shape — take it or leave it.

Happy to approve once the test helper is cleaned up.

Comment thread arrow/flight/flightsql/server_test.go Outdated
default:
err = fmt.Errorf("unknown query: %s", query)
}
if isUpdate != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The isUpdate := … declarations inside the two cases above shadow the outer var isUpdate *bool, so this if isUpdate != nil always sees nil — it's dead code. The cases already set result.IsUpdate directly, so the outer var isUpdate *bool and this block can both be removed.

@ennuite ennuite Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this review.
Addressed in 3466969

Comment thread arrow/flight/flightsql/client.go Outdated
// IsUpdate returns the server's hint about whether this prepared statement
// should be executed as an update (true) or query (false). If nil, the server
// did not provide a hint and the client can choose how to execute the statement.
func (p *PreparedStatement) IsUpdate() *bool { return p.isUpdate }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning the internal *bool leaks mutable state (a caller can do *ps.IsUpdate() = false and mutate the statement) and isn't idiomatic for an optional. Prefer IsUpdate() (val bool, ok bool), or return a copy.

@ennuite ennuite Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for pointing this out. I took the idiomatic approach with the double boolean return, see 2c8c152

@zeroshade

Copy link
Copy Markdown
Member

@ennuite are you still working on this?

@ennuite

ennuite commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

@zeroshade Sorry, I will resume work on this now and will address the comments this week.

@ennuite
ennuite marked this pull request as draft July 3, 2026 02:27
@ennuite

ennuite commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@zeroshade I addressed your review comments, and refactored some other poor patterns in the tests.

I will put this as ready for review again once I update apache/arrow-adbc#4161 with the changes I did now and ran end-to-end tests to ensure that everything is ok.

@ennuite
ennuite force-pushed the gh-705-add-is-update-field branch from a26b6b3 to f6a8588 Compare July 26, 2026 22:10
@ennuite
ennuite marked this pull request as ready for review July 27, 2026 01:08

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pushing this through — and thanks especially for the thorough test additions; the prepared-statement integration tests are a genuine gap this fills independently of the feature itself.

I verified the change fairly aggressively and the feature itself is correct. Summary of what I checked:

Check Result
Spec conformance vs upstream apache/arrow format/FlightSql.proto optional bool is_update = 4; doc comments match the spec verbatim
Upstream status apache/arrow#49498, merged 2026-06-09 — canonical, not a proposal
Generated-file semantic delta (all wire tags / exported methods / types / enum constants, vs true merge-base) Exactly +1 wire tag, +1 getter, 0 type changes, 0 enum changes
Runtime descriptor #4 is_update kind=bool hasPresence=true
Presence round-trip true, false, and unset all round-trip correctly — importantly false survives the wire, confirming real presence semantics
go build / go vet / go test ./arrow/flight/... All clean; the 4 new tests confirmed executing

So I want to be clear that the -868 net lines in FlightSql.pb.go is not hiding anything — I checked that specifically.

I'm marking this request changes for one substantive reason only: the unset/nil case is never asserted anywhere. Every new test asserts ok == true, but ok == false is the backward-compatibility contract with the (currently overwhelming majority of) servers that don't send the hint. That's the one path most likely to regress silently and it's the cheapest possible test to add.

Everything else below is either a release-note item or a nit. Two design choices I want to explicitly endorse so they don't get second-guessed in later review rounds: the (val, ok bool) getter shape, and keeping the hint purely informational. Details inline.

s.Equal(&emptyFlightInfo, info)
}

func (s *FlightSqlClientSuite) TestPreparedStatementExecuteWithIsUpdateFalse() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test for the unset/nil case. This is my only blocking request.

Both new client tests here, and both new server tests, assert ok == true. Nothing anywhere asserts ok == false. That's the path taken by every server that doesn't implement the new field — i.e. the backward-compatibility contract — and it's currently the only untested branch of the getter.

TestPreparedStatementExecute above (line 368) already builds an ActionCreatePreparedStatementResult with no IsUpdate, so this can be as small as adding two lines there:

val, ok := prepared.IsUpdate()
s.False(ok)
s.False(val)

Worth locking down explicitly because a future refactor that switched isUpdate *bool to a plain bool would silently turn "no hint" into "is a query" — which is a real behavior change for clients that branch on this — and no existing test would catch it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want the s.False(val) check? If s.False(ok), then val is meaningless.

@ennuite ennuite Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I addressed this in eb16088

I do not think it makes sense to have the check in val: it being False seems like an implementation detail, and not a part of the contract of the IsUpdate() method. Let me know if you disagree: I can change it to look exactly like your suggestion.

handle: result.PreparedStatementHandle,
datasetSchema: dsSchema,
paramSchema: paramSchema,
isUpdate: result.IsUpdate,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two small things on this propagation site.

  1. Untested. TestPreparedStatementLoadFromResult (client_test.go:971) is the only coverage for LoadPreparedStatementFromResult, and it doesn't assert IsUpdate. Since that test already constructs the result and checks the other two propagated fields (ParameterSchema, DatasetSchema), adding an IsUpdate assertion there would be consistent and nearly free. (Couldn't leave this as an inline comment on that line — it's outside the diff.)

  2. Nit, no change required: this is the one place that stores a *bool owned by the caller, so a caller mutating result.IsUpdate after this call would change the statement's view. Prepare/PrepareSubstrait don't have this property since they point at their own unmarshaled local. Not worth a defensive copy unless you'd want it for handle too — just noting it so the asymmetry is a conscious choice.

@ennuite ennuite Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I addressed this with 2 tests. In TestPreparedStatementLoadFromResult I let things as they are, and tested with the new field unset, see 8ba207b. Note that this test has the same nuance about val that I pointed out in feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult #732 (comment)

I added a new test that checks that the field is properly loaded also when set, see a334b86

  1. For that one I purposefully followed the example set by handle, but I will admit I had my own reservations while doing it.

I will not change this in this PR unless you ask me to, but in your opinion is it worth a follow-up PR to have a defensive copy for both handle and isUpdate? If so I can tackle that after this one

Comment thread arrow/flight/flightsql/client.go
ParameterSchema *arrow.Schema
// IsUpdate indicates whether the prepared statement should be executed
// as an update (true) or query (false). If nil, the client can choose.
IsUpdate *bool

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a defect, but this needs a release note, because the public API impact is wider than this struct.

types.go:882 has:

type CreatePreparedStatementResult = pb.ActionCreatePreparedStatementResult

Being a type alias, the regenerated proto field lands on a public flightsql symbol too. So this PR adds an exported field to two publicly-reachable structs. Keyed composite literals are unaffected (and that's what I found in the downstream usage I spot-checked), but any downstream unkeyed literal like:

return flightsql.ActionCreatePreparedStatementResult{handle, ds, ps}, nil

will stop compiling. That's permitted under the Go 1 compatibility rules — unkeyed literals of imported struct types are explicitly not covered — and go vet's composites check flags them, so real-world exposure should be low. I don't think it should block the PR, but it's worth a line in the changelog rather than letting someone discover it at upgrade time.

The unconditional result.IsUpdate = output.IsUpdate on both the statement and Substrait branches is correct, by the way — both sides are *bool, so nil flows through and no presence check is needed. Good that the Substrait path wasn't forgotten.

@ennuite ennuite Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment mentions 2 distinct public structs. For clarity, you are asking about release notes for both of them, right?

I updated the PR description with the breaking change. I looked into how release notes are generated, and it seems like at release time Github will use the PR title in the release notes, but not the description. Given that the description is the place where the breaking change is highlighted, do I need to add release notes elsewhere?

Explanation of the 2 structs

If you agree with my PR description you don't need to read this detail, I leave it for clarity in case I misunderstood your ask.

The type alias refers to flightsql.CreatePreparedStatementResult. This is the raw proto version, what goes in the wire, and I think it's there so that users of the flightsql package don't need to import the proto package directly. In this codebase, this is only used in LoadPreparedStatementFromResult()

But the line about the unkeyed literal is referring to a different struct flightsql.ActionCreatePreparedStatementResult that is unrelated to the alias. It is defined in server.go, and it is a developer-friendly type with the *arrow.Schema objects decoded from wire format. It is used in quite a few places in the codebase.

Both of them fall under the situation of a public struct where unkeyed literals will break, so I put both in the PR description.

Thank you for pointing out the compatibility rules, for reference I found them at https://go.dev/doc/go1compat:

For the addition of features in later point releases, it may be necessary to add fields to exported structs in the API. Code that uses unkeyed struct literals (such as pkg.T{3, "x"}) to create values of these types would fail to compile after such a change. However, code that uses keyed literals (pkg.T{A: 3, B: "x"}) will continue to compile after such a change. We will update such data structures in a way that allows keyed struct literals to remain compatible, although unkeyed literals may fail to compile.

Comment thread arrow/flight/gen/flight/FlightSql.pb.go
Comment thread arrow/flight/flightsql/server_test.go
Comment thread arrow/flight/flightsql/server_test.go
@ennuite

ennuite commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the very thorough review!

I'm going over the comments one by one, I might be a bit slow as I'm ramping up on some of the Go concepts, and your feedback has been a great learning experience.

I appreciate your patience as I improve the PR. I will continue working on it in the next few days, and at the end I'll try to tackle #1029 as well.

@zeroshade

Copy link
Copy Markdown
Member

sounds good! Just ping me here when you're ready for another review! feel free to ask any questions if you run into trouble

@ennuite

ennuite commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Still not ready for a review as I still need to look into the protobuf stuff and the release notes, but a question popped in my mind (which is probably nitpicky).

What is your opinion on s.Require.False() vs s.False()? I haven't found a canonical way to handle these (although I understand in how their behavior differs).

For the server tests, I'm always using Require() for every check of values returned by PreparedStatement.IsUpdate(), because the code after those checks (if there is any) just execute the query, and without proper values returned by IsUpdate we don't know how to properly execute the query (DoPut vs DoGet)

For the client tests:

  1. In my new tests (TestPreparedStatementExecuteWithIsUpdateFalse and TestPreparedStatementExecuteUpdateWithIsUpdateTrue) I also use Require() using the same logic as above.
  2. For TestPreparedStatementLoadFromResultWithIsUpdate: given that we are just testing the loading and not execution, it would be interesting to see if ok was loaded incorrectly and val correctly, so I don't use Require()
  3. For the pre-existing tests I don't use Require because there are interesting things to test even if these fields are broken (the tests were not created solely for this feature).

@zeroshade

Copy link
Copy Markdown
Member

What is your opinion on s.Require.False() vs s.False()? I haven't found a canonical way to handle these (although I understand in how their behavior differs).

Generally my way of looking at it is pretty much what you describe, whether I find the subsequent tests still interesting/useful if the condition I'm asserting fails. Essentially if I'm testing a condition that if it fails I know that every subsequent assertion in the test will fail, then I just use Require so that it bails out early on the first failure. If I just want to assert something but it's still useful for diagnostic (or other test cases) for the test to continue, then I don't use Require

@zeroshade
zeroshade marked this pull request as draft August 10, 2026 16:36
@zeroshade

Copy link
Copy Markdown
Member

@ennuite Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

  • Unaddressed Copilot review — an unresolved review thread from Copilot has been open since 2026-06-08 with no reply (thread). See docs.

Note: Your branch is 109 commits behind main. Please rebase and push again to get up-to-date CI results.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.


Note: This comment was drafted by an AI-assisted triage tool run by a maintainer, and may contain mistakes. Once you have addressed the points above, an Apache Arrow Go maintainer — a real person — will take the next look at your PR. If anything here looks wrong, say so on the PR and a maintainer will follow up. See CONTRIBUTING.md for the project's contribution conventions.

@ennuite
ennuite force-pushed the gh-705-add-is-update-field branch from 3009533 to c1bc900 Compare August 15, 2026 18:47
@ennuite
ennuite marked this pull request as ready for review August 15, 2026 20:32
@ennuite
ennuite marked this pull request as draft August 15, 2026 20:36
The rebase brought in 9e6dadb which now clears the FlightDescriptor
after the schema message, so the record batch send has a nil descriptor.
Update TestPreparedStatementExecuteUpdateWithIsUpdateTrue to expect two
separate Send calls matching that new behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ennuite
ennuite marked this pull request as ready for review August 15, 2026 21:23
@ennuite

ennuite commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Addressed all comments and rebased @zeroshade. Let me know what you think!

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for all the work here and getting this in!

@zeroshade
zeroshade merged commit 3488b5b into apache:main Aug 17, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FlightSQL] - Add new is_update field to PreparedStatement

3 participants